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Executive  Summary: 

In  phase  I  of  the  Yale/Plain  Sight  Systems  project  we  have  concentrated  on  dynamic 
sensing  and  processing  for  hyperspectral  imaging. 

•  We  built  two  digitally  controlled  (using  digital  mirror  array  MOEMS)  sensing 
systems  for  direct  manipulation  of  light  streams  in  both  passive  and  active  sensing 
modes. 

•  In  the  active  mode  we  assembled  a  tabletop  environment  for  testing  and 
demonstration  of  massive  data  acquisition  and  handling.  Virtually  all  aspects  of 
the  ISP  concept  can  be  emulated  in  this  environment,  where  imaging  is  done 
through  a  microscope,  dermascope,  or  endoscope. 

To  be  specific  the  components  of  the  system  consist  of 

•  A  digitally  controlled  spectrally  tunable  light  source  capable  of  generating  any 
desired  spectral  profile  of  light  in  the  range  of  400-900  nm.  The  light  spectrum 
can  be  encoded  by  pseudo  random  code  to  optimize  SNR.  The  light  can  be  tuned 
to  reveal  specific  targets  and  suppress  clutter  in  a  complex  environment.  The 
tuning  of  the  light  is  integrated  in  real  time  with  detection  and  identification  tasks. 

•  The  data  acquisition  test  platform  is  a  microscope  on  which  the  scenes  and 
targets  consist  of  histology  microdot  slides  of  hundreds  of  tissue  samples.  The 
tuned  light  source  is  used  as  the  microscope  light  source,  to  allow  acquisition  of 
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full  hyperspectral  datacubes.  A  similar  platform  was  built  for  direct  monitoring  of 
skin  lesions  using  a  dermascope. 

•  As  an  example  of  interactive  sensing  and  processing  in  this  environment, 
we  start  with  the  expert  (pathologist)  pointing  out  a  few  cancer  cells  (in  a  small 
region).  These  are  “diffused  mathematically”  to  other  very  similar  spectral  or 
spatio/spectral  objects  in  the  scene.  The  expert  points  out  obvious  failures,  and  this 
knowledge  introduces  new  features  to  refine  the  process  by  looking  for  new 
spectro/spatial  features . 

The  light  is  then  tuned  so  that  each  measurement  by  a  pixel  of  the  camera  measures 
directly  a  spectral  feature  of  the  tissue  at  that  location.  If,  for  example,  we  display  the 
top  three  features  in,  say,  RGB,  the  potential  cancer  cells  are  highlighted  in  the  other 
portion  of  the  scenes  (the  rest  of  the  slide  or  other  slides). 

This  can  be  realized  at  video  rates,  since  we  only  measure  information  of  interest  (and 
not  a  whole  spectrum).  The  conventional  mode  here  would  be  to  collect  the  whole 
spectral  cube,  digitize,  extract  features  and  display.  Here  we  are  only  measuring 
directly  information  prior  to  digitization,  avoiding  the  raw  data  glut. 

•  A  variety  of  analytical  tools  for  achieving  the  diffusion,  as  well  as  low 
dimensional  parameterizations  of  geometric  structures  in  the  hyper  spectral  data 
have  been  developed.  These  tools  enable  quantification  of  data  features 
probability  density  estimation,  detection  of  outliers  and  many  other  signal 
processing  tasks  on  line  . 

The  advantage  of  tissue  data  is  that  it  is  as  varied  as  any  landscape,  with  targets  which  are 
elusive  and  abstract,  for  which  conventional  template  matching  techniques  fail.  We  were 
forced  to  develop  nonlinear  variants  of  principal  components  analysis  to  help  in  the 
dimensional  reduction  information  extraction  tasks.  These  methods  which  involve  a 
blend  of  diffusion  theory,  spectral  theory,  geometry  and  harmonic  analysis,  have  also 
been  validated  in  the  context  of  characterization  of  inertial  manifolds  for  nonlinear 
dynamics  where  the  data  is  generated  by  dynamical  systems  and  the  inertial  manifolds 
are  obtained  empirically  from  computer  experiments  leading  to  identification  and  explicit 
parameterization  of  relevant  observables. 

•  In  the  process  described  above  we  are  using  active  interrogation  of  the  target  with 
specifically  designed  light  so  that  the  interaction  with  the  material  and  the  light 
will  reveal  information  concerning  the  target. 

•  We  have  also  considered  the  dual  problem  in  which  the  data  acquisition  is  passive 
but  the  light  streams  are  digitally  controlled  before  sensing. 

•  We  have  built  a  coded  aperture  spectral  imaging  camera.  The  purpose  of  the 
camera  is  to  detect  and  reveal  camouflaged  objects,  anomalies  in  clutter,  chemical 
signatures  etc. 


By  imaging  first  on  a  digital  mirror  array,  prior  to  spectral  separation,  the  camera  controls 
both  the  spatial  region  whose  spectral  features  are  measured  as  well  as  the  corresponding 
combination  of  spectral  features.  This  system  is  not  as  versatile  as  the  active  system  but 
opens  interesting  possibilities  if  combined  with  digitally  tunable  CMOS  imaging  arrays. 

•  The  camera  described  above  has  been  redesigned  in  the  Near  Infra  Red,  as  a 
prototype  for  field  testing  by  Lockheed  Martin.  This  will  enable  evaluation  of 
various  algorithmic  and  driving  tradeoffs  in  operational  situations. 


We  provide  a  few  examples  of  the  ideas  described  above,  and  details  are  provided  in  the 
appended  documents. 

The  process  of  tuning  the  spectral  acquisition  to  detect  a  target  is  illustrated  in  the 
following  images,  where  the  target  in  the  first  image  is  extracted  by  measuring  directly 
the  residual  spectral  energy  over  the  majority  average  clutter  spectrum  ,  and  the  displayed 
in  rgb. 


Filtered  Image 
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Fake  vegetation  is  discovered  easily  through  the  standoff  coded  aperture  imager,  Bottom  right;  a  conventional 
spectrograph  image  under  same  light  integration  setting. 


The  control  of  the  light  spectra,  either  on  acquisition  or  in 
active  mode  for  probing  is  achieved  using  a  TI  Digital 
mirror  array  described  below.  By  directing  light  rays  onto  a 
grating  we  control  the  amount  of  light  going  through  the 
system  in  any  frequency  band. 


Mirrors  are 
built  on  top 
of  integrated 
circuits  and 
assembled 
in  a  grid-like 
formation 


Mirrors  rotate  along  their  diagonal 
axis  by  exactly  ±  10°,  which  is  what 
makes  the  DMA  a  digital  devise 

•  0  =  no  light  reflected  (-10°) 

•  1  =  all  light  reflected  (+10°) 


A  complete  DMA 
contains  848 
columns  and  600 
rows  of  mirrors 
and  measures 
10.2  mm  x  13.6 
mm.  Here,  a 
DMA  is  shown 
with  its  glass 
cover  removed. 
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A  digitally  controlled  tuned  light  source  can  be  used  as  a 
spectrometer  and/or  direct  chemometric  analysis  system. 

By  tuning  the  illumination  spectrum  to  interact  with  the 
absorption  response  of  material  in  the  scene  ,we  can  directh 
measure  the  presence  of  specific  objects. 


We  have  also  developed  a  passive  DMD  based  system 
which  produced  the  results  on  camouflaged  objects 
described  above . 


For  this  system  we  have  demonstrated  that  by  using  pseudo 
random  encoding  of  the  spectral  acquisition  process  we 
enable  a  superior  performance  of  the  camera,  and  that  in 
particular  we  should  be  able  to  collect  spectral  data  at  video 
rates  , while  collection  in  a  conventional  raster  mode  fails. 


Shutter  speed  (sec.  log  scale) 


The  hardware  devices  above  have  been  extensively  tested 
A  software  hyperspectral  explorer  tool  kit  has  been 
developed  as  an  algorithm  exploration  tool  box  . 

(See  appendix  A ) 


